should be around 0.5 (between -1 and 1 according to Gunnar, but that seems just wrong)
generalisation of Brier score to continuous variables. Smaller is better.
Advantage: logarithmic Score penalises underestimating uncertainty heavily. I feel this is what we want.
Drawback: In contrast to the CRPS, the computation of LogS requires a predictive density. An estimatorcan be obtained with classical nonparametric kernel density estimation (KDE, e.g. Silverman1986). However, this estimator is valid only under stringent theoretical assumptions and canbe fragile in practice: If the outcome falls into the tails of the simulated forecast distribution,the estimated score may be highly sensitive to the choice of the bandwidth tuning parameter.In an MCMC context, a mixture-of-parameters estimator that utilizes a simulated sampleof parameter draws rather than draws from the posterior predictive distribution is a better
–> especially problematic I think if we work with traces and only small sample sizes?
Question do we now the posterior distribution of our draws?
1 day ahead prediction
Across the bench, Sparse AR seems the most reasonable (little bias, ok DSS and LogS)
AR1 seems to be very unconfident (and therefore performs well on LogS) AR1 seems to be downward biased.
1 day ahead metrics
| horizon | score | model | bottom | lower | median | mean | upper | top | sd |
|---|---|---|---|---|---|---|---|---|---|
| 1 | bias | AR 1 | 0.0000000 | 0.1800000 | 0.4000000 | 0.4208571 | 0.6400000 | 0.9600000 | 0.2805722 |
| 1 | bias | Semi-local linear trend | 0.0045000 | 0.2600000 | 0.5000000 | 0.4886122 | 0.7000000 | 1.0000000 | 0.2828632 |
| 1 | bias | Sparse AR | 0.0000000 | 0.2600000 | 0.4400000 | 0.4429796 | 0.6000000 | 0.9400000 | 0.2539925 |
| 1 | bias | Student local linear trend | 0.0000000 | 0.2800000 | 0.4600000 | 0.4940408 | 0.7400000 | 1.0000000 | 0.2887849 |
| 1 | crps | AR 1 | 0.0097440 | 0.0293893 | 0.0526451 | 0.0794650 | 0.0890898 | 0.3660674 | 0.0898345 |
| 1 | crps | Semi-local linear trend | 0.0084990 | 0.0180820 | 0.0310165 | 0.0582366 | 0.0626547 | 0.2951930 | 0.0804311 |
| 1 | crps | Sparse AR | 0.0101793 | 0.0239739 | 0.0382058 | 0.0670498 | 0.0746289 | 0.3242686 | 0.0869147 |
| 1 | crps | Student local linear trend | 0.0078844 | 0.0175479 | 0.0276261 | 0.0574862 | 0.0627260 | 0.3423243 | 0.0799965 |
| 1 | dss | AR 1 | -6.7070804 | -4.2726994 | -3.2469240 | -2.7868620 | -2.0327237 | 4.1355028 | 3.5341067 |
| 1 | dss | Semi-local linear trend | -6.8608629 | -5.4140323 | -4.4642409 | -3.1949170 | -3.0414601 | 6.9085930 | 7.2741175 |
| 1 | dss | Sparse AR | -6.2707872 | -4.7646250 | -4.0417495 | -3.2725326 | -2.6254400 | 3.2154416 | 4.1854044 |
| 1 | dss | Student local linear trend | -6.9603421 | -5.6213290 | -4.5726281 | -3.0854304 | -3.0623496 | 8.5178083 | 8.6348196 |
| 1 | logs | AR 1 | -2.3759070 | -1.2771117 | -0.8254659 | -0.3512798 | -0.2124687 | 3.7791279 | 3.3550183 |
| 1 | logs | Semi-local linear trend | -2.4113148 | -1.7454975 | -1.2850647 | 0.1676840 | -0.5942576 | 5.1365309 | 14.1584179 |
| 1 | logs | Sparse AR | -2.2151640 | -1.4355747 | -1.0007693 | -0.4215960 | -0.4075420 | 2.3876052 | 4.8899705 |
| 1 | logs | Student local linear trend | -2.5628558 | -1.8588923 | -1.3665089 | 0.6381316 | -0.5829726 | 7.2153839 | 21.4649779 |
| 1 | sharpness | AR 1 | 0.0221096 | 0.0709010 | 0.1052553 | 0.1373040 | 0.1700109 | 0.4080446 | 0.1137602 |
| 1 | sharpness | Semi-local linear trend | 0.0220793 | 0.0425032 | 0.0709581 | 0.0970184 | 0.1132235 | 0.3527295 | 0.0872100 |
| 1 | sharpness | Sparse AR | 0.0273751 | 0.0597690 | 0.1020642 | 0.1307556 | 0.1481714 | 0.4327618 | 0.1075388 |
| 1 | sharpness | Student local linear trend | 0.0201258 | 0.0389854 | 0.0597141 | 0.0908121 | 0.1074593 | 0.3644611 | 0.0861209 |
3 day ahead prediction
AR1 seems very bad in terms of bias and everything.
Sparse AR is the best in terms of crps, AR1 the worst. Sparse AR also best with dss
–> take Sparse AR
All models have a tendency to be downwards biased, the local and semilocal ones tend to do a bit better.
3 day ahead metrics
| horizon | score | model | bottom | lower | median | mean | upper | top | sd |
|---|---|---|---|---|---|---|---|---|---|
| 7 | bias | AR 1 | 0.0000000 | 0.0250000 | 0.2800000 | 0.3428108 | 0.5800000 | 1.0000000 | 0.3221013 |
| 7 | bias | Semi-local linear trend | 0.0000000 | 0.1200000 | 0.4100000 | 0.4577838 | 0.8150000 | 1.0000000 | 0.3509869 |
| 7 | bias | Sparse AR | 0.0000000 | 0.0650000 | 0.3600000 | 0.3907568 | 0.6400000 | 1.0000000 | 0.3230276 |
| 7 | bias | Student local linear trend | 0.0000000 | 0.1400000 | 0.4400000 | 0.4797297 | 0.8200000 | 1.0000000 | 0.3453366 |
| 7 | crps | AR 1 | 0.0224146 | 0.0939478 | 0.1810374 | 0.3033660 | 0.3944736 | 1.1650918 | 0.3233669 |
| 7 | crps | Semi-local linear trend | 0.0408044 | 0.1136825 | 0.2071928 | 0.3195261 | 0.4456299 | 1.1783405 | 0.3019611 |
| 7 | crps | Sparse AR | 0.0411224 | 0.0869667 | 0.1639003 | 0.2659190 | 0.3420645 | 0.9273282 | 0.2865941 |
| 7 | crps | Student local linear trend | 0.0366862 | 0.1256516 | 0.2213939 | 0.3495900 | 0.4473379 | 1.3712852 | 0.3557144 |
| 7 | dss | AR 1 | -5.1511479 | -1.9594018 | -0.9830888 | 9.6924487 | 2.0741012 | 140.8036503 | 50.5344641 |
| 7 | dss | Semi-local linear trend | -3.6826454 | -1.9456928 | -0.8493304 | 4.4328664 | 1.3250952 | 34.6275300 | 32.3297634 |
| 7 | dss | Sparse AR | -3.7060228 | -2.4520010 | -1.2548468 | 2.4033523 | 1.0483922 | 34.4984516 | 12.6638976 |
| 7 | dss | Student local linear trend | -3.6042707 | -1.6218931 | -0.7108610 | 6.2082494 | 1.2599417 | 48.4292621 | 48.2765050 |
| 7 | logs | AR 1 | -1.6846635 | -0.1356936 | 0.4478093 | Inf | 2.0985518 | 287.2871496 | Inf |
| 7 | logs | Semi-local linear trend | -0.9164213 | 0.0317563 | 0.5867681 | Inf | 1.5800334 | 188.5458002 | Inf |
| 7 | logs | Sparse AR | -0.8712581 | -0.2524683 | 0.3474695 | 4.8517056 | 1.3516859 | 43.3777326 | 24.7326588 |
| 7 | logs | Student local linear trend | -1.0542115 | 0.1465011 | 0.6724498 | Inf | 1.6191972 | 586.2568172 | Inf |
| 7 | sharpness | AR 1 | 0.0260282 | 0.1413464 | 0.2472176 | 0.3073751 | 0.3930559 | 0.9850034 | 0.2636063 |
| 7 | sharpness | Semi-local linear trend | 0.0000000 | 0.1636543 | 0.2814537 | 0.3272691 | 0.4235071 | 0.8840668 | 0.2361977 |
| 7 | sharpness | Sparse AR | 0.0601856 | 0.1622306 | 0.2385763 | 0.2984173 | 0.3420976 | 0.8644332 | 0.2239063 |
| 7 | sharpness | Student local linear trend | 0.0000000 | 0.1693309 | 0.3359365 | 0.3761931 | 0.5000669 | 1.0860822 | 0.2989137 |
7 day ahead prediction
Sparse AR still downwards biased local and semilocal doing better
crps: student local is best (it shouldn’t?), Semilocal and Sparse AR about equal. Sparse AR most consistent.
dss: student local doing best, then Sparse AR and semilocal
LogS: student local best, then Sparse AR wide ahead of the others.
Sparse AR most consistent.
3 day ahead metrics
| horizon | score | model | bottom | lower | median | mean | upper | top | sd |
|---|---|---|---|---|---|---|---|---|---|
| 3 | bias | AR 1 | 0.0000000 | 0.0800000 | 0.3200000 | 0.3870222 | 0.6400000 | 1.0000000 | 0.3241965 |
| 3 | bias | Semi-local linear trend | 0.0000000 | 0.1800000 | 0.4600000 | 0.4798667 | 0.7750000 | 1.0000000 | 0.3306509 |
| 3 | bias | Sparse AR | 0.0000000 | 0.1600000 | 0.4000000 | 0.4207111 | 0.6400000 | 0.9800000 | 0.3044165 |
| 3 | bias | Student local linear trend | 0.0000000 | 0.2200000 | 0.4500000 | 0.4877333 | 0.8000000 | 1.0000000 | 0.3268705 |
| 3 | crps | AR 1 | 0.0171687 | 0.0614331 | 0.1103108 | 0.1825217 | 0.2175170 | 0.8051064 | 0.2065734 |
| 3 | crps | Semi-local linear trend | 0.0201682 | 0.0477773 | 0.0924898 | 0.1629336 | 0.1807126 | 0.8202050 | 0.2113492 |
| 3 | crps | Sparse AR | 0.0206726 | 0.0506307 | 0.0928446 | 0.1560582 | 0.1823767 | 0.7172435 | 0.1885933 |
| 3 | crps | Student local linear trend | 0.0163728 | 0.0477265 | 0.0863423 | 0.1690528 | 0.1992893 | 0.9294194 | 0.2305773 |
| 3 | dss | AR 1 | -5.9276533 | -2.9450615 | -1.7709012 | 0.6555864 | -0.2748510 | 25.7626906 | 12.1364841 |
| 3 | dss | Semi-local linear trend | -5.1714026 | -3.4990369 | -2.2604842 | 1.0410709 | -0.5360296 | 22.6703036 | 19.6737737 |
| 3 | dss | Sparse AR | -5.1341712 | -3.3923785 | -2.2980771 | -0.6108254 | -0.7036617 | 16.5860866 | 7.2176870 |
| 3 | dss | Student local linear trend | -5.4740366 | -3.4486021 | -2.4468675 | 7.1027040 | -0.5750851 | 33.8384259 | 128.8999208 |
| 3 | logs | AR 1 | -2.0104986 | -0.6239897 | -0.0319122 | Inf | 0.7560601 | 26.4391328 | Inf |
| 3 | logs | Semi-local linear trend | -1.6811899 | -0.7726919 | -0.1932857 | Inf | 0.7073339 | 21.2044455 | Inf |
| 3 | logs | Sparse AR | -1.7106809 | -0.7202203 | -0.2495037 | 1.3074993 | 0.5516765 | 16.3867174 | 9.0427728 |
| 3 | logs | Student local linear trend | -1.8849301 | -0.7960376 | -0.2705402 | Inf | 0.7166684 | 71.4039034 | Inf |
| 3 | sharpness | AR 1 | 0.0259101 | 0.1093206 | 0.1716067 | 0.2187139 | 0.2829760 | 0.6810136 | 0.1803758 |
| 3 | sharpness | Semi-local linear trend | 0.0358389 | 0.0953194 | 0.1487791 | 0.1922980 | 0.2379244 | 0.6989025 | 0.1579592 |
| 3 | sharpness | Sparse AR | 0.0448981 | 0.1150896 | 0.1658032 | 0.2166656 | 0.2701764 | 0.6841201 | 0.1613564 |
| 3 | sharpness | Student local linear trend | 0.0255382 | 0.0886775 | 0.1449848 | 0.1864862 | 0.2316276 | 0.6289547 | 0.1665941 |